Skip to content

Serve on every shard: one reactor and SO_REUSEPORT listener each - #58

Merged
pathscale merged 2 commits into
masterfrom
feat/reuseport-shards
Sep 23, 2026
Merged

pathscale merged 2 commits into
masterfrom
feat/reuseport-shards

Conversation

@pathscale

Copy link
Copy Markdown
Owner

Fixes the root cause behind WS_SHARDS=1 in consumers' fly.toml. The server ran on one thread whatever WS_SHARDS said, and warned at every boot that a nagoya listener couldn't be shared between reactors.

Depends on pathscale/nagoya#16 (0.1.13, bind_shared/SO_REUSEPORT).

  • listen_on: with more than one shard, it binds the first listener shared, then starts a ws-shard-N thread for each extra shard. Each thread has its own Reactor::local and its own SO_REUSEPORT listener on the concrete address, so a connection is accepted, registered and served on one reactor and never crosses a thread.
  • prepare sets up the server (Arc<Self>) and the connection table once, shared by every shard, so the toolbox reaches a connection whichever shard holds it. serve_shard is the old accept loop.
  • The first shard owns the stop (signals or the caller's) and cancels the others through Shutdown. listen returns after all shards have joined.
  • One shard: same path as before.
  • TcpListener::bind_shared / bind_any_shared. The docs on listen and the accept loop are updated.
  • 3.3.0 to 3.3.1, and nagoya's minimum goes to ^0.1.13 because bind_shared requires it.

Tested: new tests/sharded_listen.rs, a WS_SHARDS=3 server over TCP. 12 clients each completed an echo request, served by the first shard and ws-shard-2 (macOS doesn't balance; Linux does), and one stop ended all shards with listen_until returning Ok. Full suite: 151 pass. Clippy is clean with --all-features. Built against the local nagoya branch.

Once both are released, consumers can drop WS_SHARDS from fly.toml.

The server ran on one thread whatever WS_SHARDS said, and warned about it at
every boot, because a nagoya listener could not be shared between reactors.
Consumers set WS_SHARDS=1 to hide the warning.

nagoya 0.1.13 adds bind_shared (SO_REUSEPORT). listen now starts a thread per
extra shard, each with its own local reactor and its own listener on the one
port, so a connection is accepted, registered and served by the same reactor
and never crosses a thread. The first bind picks the address and the others
take it exactly, which matters when the port was zero. The server and the
connection table are prepared once and shared, so the toolbox reaches a
connection on any shard. The first shard owns the stop, whether a signal or
the caller's, and cancels the rest; listen returns once all have joined.

With one shard nothing changes. tests/sharded_listen.rs drives three shards
over TCP: every client completes a request, and one stop ends them all.
Release as 3.3.1, needing nagoya ^0.1.13.
pathscale pushed a commit to pathscale/api.support.cafe that referenced this pull request Sep 23, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked.
endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves
every shard on its own reactor with an SO_REUSEPORT listener, so the default,
the CPU count, is what runs.
@pathscale
pathscale merged commit d59244e into master Sep 23, 2026
@pathscale
pathscale deleted the feat/reuseport-shards branch September 23, 2026 22:46
pathscale pushed a commit to pathscale/api.support.cafe that referenced this pull request Sep 24, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked.
endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves
every shard on its own reactor with an SO_REUSEPORT listener, so the default,
the CPU count, is what runs.
pathscale pushed a commit to pathscale/api.support.cafe that referenced this pull request Sep 24, 2026
It hid endpoint-libs' warning that it served one thread whatever was asked.
endpoint-libs 3.3.1 (pathscale/endpoint-libs#58, on nagoya 0.1.13) serves
every shard on its own reactor with an SO_REUSEPORT listener, so the default,
the CPU count, is what runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant